python run_generation.py --model_name_or_path Qwen/Qwen2-0.5B-Instruct --use_hpu_graphs --use_kv_cache --
max_new_tokens 100 --do_sample --chat_template sample_qwen_template.json --bf16 --batch_size 2

input 1: ('<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nGive me a short introduction to large language model.<|im_end|>\n',)
output 1: ('system\nYou are a helpful assistant.\nuser\nGive me a short introduction to large language model.\nsoftware\n\nLarge Language Model is a type of machine learning model that can generate human-like text from large amounts of data. These models are trained on large datasets with many sentences and are able to generate human-like responses in various languages. Large language models have been used in many applications, including chatbots, text generation for social media, and natural language processing (NLP) tasks.\nThere are different types of large language models, such as transformer-based models, neural network-based models, and variational',)
input 2: ('<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nGive me a short introduction to large language model.<|im_end|>\n',)
output 1: ('system\nYou are a helpful assistant.\nuser\nGive me a short introduction to large language model.\nuser: What is the definition of an AI?\nuser: Can you describe the process of training an AI model?\nuser: How does a deep learning algorithm learn from data?\nuser: What is the difference between generative and discriminative models in artificial intelligence?\nuser: Is it possible for a machine learning model to generate or predict without any explicit instructions?\nuser: Could AI be used as a substitute for human teachers?',)
Stats:
---------------------------------------------------------------------------------------------------------------
Throughput (including tokenization) = 441.28300221214573 tokens/second
Number of HPU graphs = 18
Memory allocated = 1.52 GB
Max memory allocated = 1.59 GB
Total memory available = 94.62 GB
Graph compilation duration = 2.503536907985108 seconds
---------------------------------------------------------------------------------------------------------------

python run_generation.py --model_name_or_path "google/gemma-1.1-2b-it" --use_hpu_graphs --use_kv_cache --max_new_tokens 100 --do_sample --chat_template sample_gemma_template.json --bf16 --batch_size 2

input 1: ('<bos><start_of_turn>user\nWrite a hello world program<end_of_turn>\n',)
output 1: ('user\nWrite a hello world program\nimport java.util.Scanner;\n\npublic class HelloWorld {\n\n    public static void main(String[] args) {\n        Scanner scanner = new Scanner(System.in);\n\n        // Read user input\n        System.out.println("Hello, world!");\n\n        // Close the scanner\n        scanner.close();\n    }\n}\n```\n\n**Explanation:**\n\n* The code you provided is a simple Java program that demonstrates how to create and use a `Scanner` object',)

input 2: ('<bos><start_of_turn>user\nWrite a hello world program<end_of_turn>\n',)
output 1: ('user\nWrite a hello world program\n```c\n#include <stdio.h>\n\nint main()\n{\n    printf("Hello, world!\\n");\n\n    return 0;\n}\n```\n\n**Explanation:**\n\n* The program starts with the `#include <stdio.h>` line, which includes the standard input/output (stdio) library. This allows the program to use functions like `printf` and `return`.\n* The `main()` function is the entry point of the program.\n',)


Stats:
--------------------------------------------------------------------------------------------------------------
Throughput (including tokenization) = 544.5759928878326 tokens/second
Number of HPU graphs                = 14
Memory allocated                    = 5.88 GB
Max memory allocated                = 6.2 GB
Total memory available              = 94.62 GB
Graph compilation duration          = 3.638087995001115 seconds
--------------------------------------------------------------------------------------------------------------